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The  Software  Supportability  Qualitative  Aseeatment  Methodology  is  a  five 
volume  reference  set  that  provides  measures  to  aid  in  the  support  of  information  systems. 
These  manuals  are  aimed  at  improving  the  support  process  by  more  accurately  assessing  the 
capabilities  of  support  organizations,  quantitatively  measuring  the  supportability  of  fielded 
systems  and  evaluating  the  operational  readiness  of  fielded  systems. 

Volume  I,  Developing  Quality  Measures  for  Information  Systems  Support,  describes  the 
three  measures  along  with  the  model  of  information  system  support  that  the  measures  are 
designed  to  satisfy.  This  is  the  main  volume  of  the  set  and  should  be  consulted  before 
implementing  the  measures  described  in  more  detail  in  the  other  volumes. 

Volume  II,  The  Review  of  Metrics  for  Developing  an  Information  Systems  Support  Mea¬ 
surement  Framework,  provides  a  survey  and  evaluation  of  current  metrics  in  terms  of  in¬ 
formation  systems  support.  Specifically,  three  classes  of  metrics  are  reviewed:  software 
product  metrics,  life  cycle  process  metrics,  and  process  management  metrics. 

Volume  III,  Implementing  the  Software  Supportability  Measure,  provides  instructions  for 
collecting  data  for  the  measure,  compiling  the  measure  by  evaluating  the  data,  and  inter¬ 
preting  the  final  result.  The  volume  also  contains  guidelines  for  improving  the  supportabilty 
of  an  information  system  based  on  its  evaluation.  Specifically,  the  volume  contains  resource 
estimations  for  compiling  and  evaluating  the  measure,  questionnaires  for  collecting  the  re¬ 
quired  data  and  step-by-step  instructions  for  measuring  the  supportability  of  an  information 
system. 

Volume  rV,  Implementing  the  Support  Organization  Assessment  Measure,  provides  in¬ 
structions  for  collecting  data  for  the  assessment,  conducting  the  assessment,  and  interpret¬ 
ing  the  final  result.  The  volume  also  contains  guidelines  for  improving  the  capabilities  of 
a  support  organization  based  on  its  evaluation.  Specifically,  the  volume  contains  resource 
estimations  for  conducting  and  evaluating  the  assessment,  questionnaires  for  collecting  the 
required  data  and  step-by-step  instructions  for  measuring  the  capabilities  of  a  support  or¬ 
ganization. 

Volume  V,  Implementing  the  Operational  Readiness  Measure,  provides  instructions  for 
collecting  data  for  the  measure,  compiling  the  measure  by  evaluating  the  data,  and  inter¬ 
preting  the  final  result.  The  volume  also  contains  guidelines  for  improving  the  operational 
readiness  of  an  information  system  based  on  its  evaluation.  Specifically,  the  volume  contains 
resource  estimations  for  compiling  and  evaluating  the  measure,  questionnaires  for  collecting 
the  required  data  and  step-by-step  instructions  for  measuring  the  operational  readiness  of 
an  information  system. 
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1  Introduction 


The  operational  readiness  measure  for  an  information  system  is  comprised  of  factors  that  affect  an 
information  system's  operational  readiness.  Operational  readiness  is  formally  defined  as  follows: 

Operational  Readiness  is  the  ability  of  a  software  system  to  perform  its  intended 
function  upon  request,  based  on: 

•  The  current  operational  state  of  the  system 

•  The  reliability  of  the  system 

•  The  supportability  of  the  system 

Operational  readiness  addresses  such  questions  such  as  “WiU  the  system  be  up  running  when 
I  need  it?”  and  “When  I  use  the  system,  can  I  expect  correct  results?”  The  information  system 
support  organization  may  utilize  this  measure  not  so  much  to  determine  the  operational  readiness 
of  an  information  system  at  the  present  time  as  to  forecast  the  operational  readiness  of  an 
information  system  for  the  near  future.  Obviously,  if  the  information  system  is  presently  in  a 
state  of  disrepair,  the  operational  readiness  is  “red”  (or  some  other  measure  indicating  a  “down” 
state),  and  no  additional  measurements  are  necessary.  However,  there  may  be  other  situations 
in  which  we  may  wish  to  identify  a  critical  operational  readiness  state.  For  example,  a  lack  of 
adequate  resources  to  support  an  information  system  combined  with  a  large  number  of  incoming 
maintenance  requests  for  that  system  could  indicate  a  critical  state. 

As  indicated  by  the  above  definition,  there  are  three  factors  of  operational  readiness.  The 
current  state  factor  measures  components  relating  to  current  system  operation  and  maintenance 
actions.  The  reliability  factor  measures  components  relating  to  the  historical  reliability  of  the 
information  system.  The  supportability  factor  measures  components  related  to  the  supportability 
of  the  information  system.  The  supportability  factor  draws  heavily  upon  the  separately  developed 
supportability  measure. 

This  document  describes  how  to  measure  the  operational  readiness  of  an  information  system 
that  is  maintained  by  your  organization.  The  second  section  details  what  resources  in  time, 
material,  and  personnel  are  required  to  compute  this  measure.  The  following  sections  describe 
the  process  for  computing  and  interpreting  the  operational  readiness  measure. 

It  is  important  you  read  through  the  following  two  sections  (up  through  Calculating  and 
Evaluating  the  Operational  Readiness  Measure)  before  beginning  this  process.  It  is  also  im¬ 
portant  that  any  personnel  who  will  provide  data  for  this  process  (by  completing  one  or  more 
questionnaires)  do  NOT  read  the  sections  on  scoring  the  questionnaires  and  evaluating  the  results 
(Sections  4  and  5)  until  after  they  have  completed  the  questionnaires. 


2  Requirements 

Material 

Little  in  the  way  of  materials  is  required  to  compile  the  operational  readiness  measure.  Ap¬ 
pendix  C  in  this  volume  contains  the  system  questionnaire,  whose  contents  are  the  complete 
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list  of  questions  to  be  answered  for  compiling  the  measure.  All  questions  in  the  questionnsure 
must  be  answered.  The  questionnaire  should  be  photocopied  and  distributed  to  the  appropriate 
personnel  (see  next  subsection).  Appendix  D  contains  directions  and  worksheets  for  scoring  the 
questionnaires.  A  set  of  directions  and  a  worksheet  should  be  photocopied  for  each  question¬ 
naire.  Appendix  £  contains  a  worksheet  that  should  be  used  to  record  the  final  results  of  the 
operational  readiness  measurement.  This  worksheet  should  be  photocopied  and  should  be  used 
in  interpreting  and  evaluating  the  final  measurement. 


Audience 

The  careful  selection  of  the  appropriate  personnel  to  complete  the  system  questionnaires  is  critical 
to  the  success  of  the  operational  readiness  measure.  An  examination  of  the  questionnaire  should 
be  the  first  step  in  choosing  your  audience.  In  general,  the  questionnaire  questionnaire  should  be 
completed  by  personnel  directly  tasked  with  the  actual  maintenance  of  the  information  system. 

Because  a  portion  of  the  questionnaire  requires  subjective  analysis  or  estimates  (in  cases  where 
raw  data  is  not  available),  distributing  the  questionnaires  to  more  than  person  involved  with 
maintaining  the  information  system  and  averaging  the  scores  should  reduce  bias  accompanying 
subjective  responses. 

Selecting  a  coordinator  to  distribute,  collect,  validate,  and  score  the  questionnaires  is  required. 
The  coordinator  is  responsible  for  distributing  the  questionnaires  and  answering  any  remaining 
questions  that  the  respondents  may  have.  The  coordinator  must  also  collect  the  questionnaire, 
verifying  that  all  questions  have  been  answered  completely.  The  coordinator  must  also  validate 
the  questionnaires  against  each  other.  Essentially  the  coordinator  assures  that  the  answers  make 
sense  (i.e.  percentages  add  up  to  100)  and  that  the  respondents  interpreted  the  questions  in  the 
same  manner.  More  information  on  this  process  can  be  found  in  the  next  section.  Finally  the 
coordinator  is  the  best  person  to  be  responsible  for  scoring  the  questionnaire  and  compiling  the 
final  results.  The  coordinator  may  NOT  complete  any  of  the  questionnaires  as  a  respondent. 

In  summary,  this  process  requires  a  minimum  of  two  personnel  to  answer  the  system  ques¬ 
tionnaire  and  to  serve  as  coordinator,  respectively.  The  final  results  will  be  more  meaningful  if 
more  than  one  person  completes  the  system  questionnaire. 


Time 

The  amount  of  time  required  to  compile  the  operational  readiness  measurement  depends  on  two 
factors;  the  amount  of  on-line  or  easily  accessible  data  and  the  number  of  personnel  tasked  to 
complete  the  system  questionnaire.  The  questionnaire  should  take  from  4  person-hours  to  12 
person-hours  to  complete.  The  amount  of  time  required  to  actually  complete  the  questionnaire 
depends  on  the  availability  and  accessibility  of  system  data.  The  questionnsdre  should  take  one 
half  of  a  person-hour  to  score. 

The  amount  of  time  required  of  the  coordinator  is  determined  by  the  number  of  personnel 
filling  out  the  questionnaire.  Validating  the  questionnaire  should  take  approximately  one  person- 
hour  per  completed  questionnaire.  The  effort  should  be  less  for  a  small  number  of  questionnaires. 

A  rough  formula  to  calculate  the  time  required  for  collecting  the  data  and  calculating  the 
measure  is  given  below. 
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SW  •  weight  based  on  accasslbility  of  information  for  tha  system. 
Range  from  1  (readily  accessible  Information)  to  3. 

SN  ■  Humber  of  systems  quest iozmaires  completed. 


TT  ■  Total  time  required  In  person-bours . 

TT  ■  4*SH*SH  +  (SH)  ♦  .Se<SH) 

I  I  I 

I  I  -  scoring 

I  I 

I  -  validation 

I 

■*  completing  system  questionnaires 


3  Calculating  and  Evaluating  the  Operational  Readiness  Mea¬ 
sure 

This  process  consists  of  six  steps. 

1.  Select  respondents  and  coordinator. 

2.  Review  questionnaire  (optional). 

3.  Fill  out  que8tionnaire(8). 

4.  Validate  que8tionnaire(8). 

5.  Score  que8tionnaire(8),  compute  measure. 

6.  Interpret  final  result. 

First,  the  personnel  tasked  with  completing  the  questionnaires  and  the  overall  coordinator 
should  be  selected.  Refer  to  the  earlier  section  for  guidelines  for  selecting  questionnaire  respon¬ 
dents.  The  system  questionnaire  included  in  this  volume  is  the  same  questionnaire  that  is  used 
to  compile  the  supportability  measure  for  an  information  system  (see  Volume  III).  Whereas  for 
the  supportability  masure  only  certain  questions  in  this  questionnaire  axe  completed,  repondents 
must  complete  all  questions  in  this  questionnaire  for  the  operational  readiness  measure. 

Optionally,  after  the  respondents  have  been  selected,  a  meeting  could  be  held  to  review  the 
questionnaire.  This  meeting  should  be  led  by  the  coordinator.  The  purpose  of  this  meeting  is 
to  assure  that  the  respondents  understand  the  questions  in  the  same  manner.  Discussions  about 
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possible  answers  should  not  be  permitted.  Only  definitional  information  should  be  distributed. 
The  advanta  of  this  meeting  is  that  it  should  quicken  the  completion  of  the  questionnaires  by 
the  respondents  and  it  should  reduce  the  variability  in  their  interpretation  of  the  questions. 

Next,  the  system  questionnaire  should  be  completed.  The  coordinator  should  be  available 
to  answer  questions  of  interpretation.  Respondents  should  be  encouraged  to  write  comments 
concerning  interpretation  next  to  their  answers.  This  effort  will  aid  the  coordinator  in  validating 
the  questionnaires. 

The  completed  questionnaire(s)  should  be  returned  to  the  coordinator  who  should  attempt 
to  validate  the  responses.  First,  all  questions  should  be  answered!  Second,  responses  contain¬ 
ing  quantitative,  non-sub jective  data  should  correspond  closely  if  not  be  equivalent.  Third,  the 
coordinator  should  look  for  differing  interpretations  by  examining  comments  duJed  by  the  re¬ 
spondents. 

Next,  the  coordinator  should  score  the  que8tlonnaire(8).  Refer  to  the  next  section  and  ap¬ 
pendices  D  and  E  for  scoring  directions.  The  system  questionnaire  scoring  directions  contains 
directions  for  scoring  the  questions,  al)  of  which  must  be  answered  to  compute  the  operational 
readiness  measure. 

If  more  than  one  questionnaire  is  completed  for  an  information  system,  the  coordinator  should 
average  the  scores  to  compute  a  final  score.  Finally,  the  coordinator  and  other  personnel  should 
consult  Section  5  in  order  to  interpret  the  final  result. 


4  Scoring  the  Questionnaire  Answers 

Appendices  D  and  E  contain  directions  for  scoring  the  individual  questionnaires  smd  computing 
the  final  measure.  Essentially,  each  answer  will  correspond  to  a  certtun  number  of  points.  Scoring 
directions  for  each  question  are  provided  with  possible  ranges  specified.  The  results  for  each 
questionnaire  are  then  recorded  on  the  questionnaire  worksheets.  The  worksheets  divide  the 
responses  into  three  categories:  Supportability,  Reliability,  and  Current  State.  These  categories 
are  the  three  major  factors  of  the  operational  readiness  measure.  The  columns  for  each  category 
should  be  totaled.  If  more  than  one  questionnaire  was  completed  for  a  given  information  system, 
the  column  totals  for  the  respective  questionnaires  should  then  be  averaged  by  the  number  of 
system  questionnaires  completed  for  the  information  system.  These  averages  should  be  recorded 
on  the  Operational  Readiness  Worksheet.  Instructions  on  the  Operational  Readiness  Worksheet 
should  then  be  followed  to  compute  the  final  result. 


5  Interpreting  the  Results 

Operational  readiness  is  a  measure  of  the  system  to  operate  correctly  on  demand.  The  measure 
is  predictive,  indicating  the  ability  of  the  information  system  to  operate  correctly  on  demand  in 
the  near  future.  Thus,  operation  readiness  depends  on  the  “current  state”  of  the  maintenance  of 
the  system  as  well  as  system  reliability.  In  addition,  operational  res^iiness  is  heavily  dependent 
upon  the  supportability  of  a  system,  since  a  system  which  is  difficult  to  support  is  more  likely 
to  have  operational  difficulties  when  the  need  for  expected  or  unexpected  maintenance  actions 
arises.  Factors  affecting  the  operational  readiness  of  an  information  system  are  therefore  divided 
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into  three  categories:  supportability,  reliability,  and  current  state.  The  operational  readiness 
measure  is  calculated  such  that  supportability  has  about  twice  as  much  influence  on  the  final 
value  of  the  measure  as  the  other  two  categories  combined. 

The  Operational  Readiness  Measure 

The  operational  readiness  measure  for  an  information  system  can  be  interpreted  in  more  than 
one  way.  First,  it  can  be  interpreted  as  the  percentage  chance  that  a  system  will  be  “opera¬ 
tionally  ready”  over  the  next  period  of  time  (a  period  of  a  month  or  less,  but  the  period  may 
vary  depending  upon  the  frequency  of  maintenance  activities  conducted  on  the  system).  This 
interpretation  is  rather  fine-grained,  however,  and  it  may  be  difficult  to  interpret  accurately. 

An  alternative  interpretation  would  be  to  assign  a  label  to  operational  readiness  based  upon 
the  final  score.  One  interpretation  of  operational  readiness  (an  interpretation  that  hcis  been  used 
to  assess  the  operational  readiness  of  hardware  equipment)  is  the  use  of  the  values  “red,”  “yellow,” 
and  “green.”  A  value  of  “red”  indicates  that  the  system  is  in  a  “down”  state;  that  is,  the  system 
is  currently  not  operational  (at  least  as  intended),  or  it  will  not  be  operational  in  the  near  future. 
A  value  of  “yellow”  indicates  that,  while  the  system  is  operational,  there  are  significant  problems 
that  may  threaten  to  bring  the  system  down  in  the  near  future.  A  value  of  green  indicates  that 
the  system  is  up  an  running  as  expected  with  no  significant  problems  likely  in  the  near  future. 

If  we  view  operational  readiness  as  a  combination  of  the  three  factors  of  supportability,  relia¬ 
bility,  and  current  state,  and  we  scale  the  total  of  all  scores  to  fit  within  the  range  of  0-100,  then 
the  following  interpretation  of  operational  readiness  may  be  useful: 

71-100  Green 
31-70  Yellow 
0—30  Red 

The  Supportability  Factor 

An  alternative  approach  of  operational  readiness  is  to  examine  the  supportability  factor  and  to 
examine  the  combination  of  the  reliability  and  current  state  factors.  (Note  that  because  the  value 
assigned  to  supportability  is  approximately  equal  to  the  combined  value  assigned  to  reliability 
and  current  state,  we  treat  reliability  and  current  state  together).  The  final  raw  score  for  the 
supportability  factor  can  range  from  2  to  150.  A  useful  interpretation  for  operational  readiness 
from  this  perspective  is  as  follows: 

106-160  Green 
45-104  Yellow 
2-44  Red 

The  criteria  that  comprise  the  supportability  factor  of  operational  readiness  include  the  size, 
age,  complexity  and  modularity  of  the  source  code,  modification  history,  existence  and  adequacy 


of  system  documentation,  existence  of  debugging  code  and  adequate  testing,  length  of  support  of 
the  system,  and  the  size  and  urgency  of  change  requests. 

Depending  on  the  severity  of  the  rating,  the  following  actions  may  be  taken  to  improve  the 
supportability  factor.  This  list  does  not  represent  all  possible  appropriate  actions. 

•  Re-design  the  system  (reverse  en^neering). 

•  Create  or  update  the  system  documentation. 

•  Increase  time  spent  on  system  testing. 

•  Accept  fewer  emergency  change  packages. 

•  Accept  fewer  large-scale  changes. 

•  Consistently  use  regression  testing. 

•  Increase  the  number  and  training  of  people  supporting  this  system. 

•  Review  the  adequacy  of  software/hardware  platform  support. 

Reliabilty  and  Current  State  Factors 

Another  view  of  operational  readiness  is  through  the  combination  of  the  reliability  and  current 
state  factors.  The  combined  raw  score  values  for  these  two  measures  can  range  from  0  to  150.  A 
useful  interpretation  from  this  perspective  follows: 

106-150  Green 
45-104  Yellow 
0—44  Red 

The  criteria  comprising  the  current  state  and  reliability  factors  include  historical  as  well  as 
current  values  of  such  items  as  number  of  maintenance  requests  already  fulfilled  or  to  be  fulfilled, 
proportion  of  corrections  and  emergency  requests,  frequency  of  contact  with  users,  number  of  staff 
currently  supporting  this  system,  and  the  estimated  effort  required  to  complete  current  changes 
to  the  information  system. 

The  combined  rating  of  reliability  and  current  state  can  vary  considerably  from  time  to  time, 
since  maintenance  requests  are  unlikely  to  originate  at  a  uniform  rate.  Depending  on  the  severity 
of  the  rating,  the  following  actions  may  be  taken  to  improve  the  current  state  and  reliability 
factors.  This  list  does  not  represent  all  possible  appropriate  actions. 

•  Increase  number  of  staff  to  support  this  system,  at  least  temporarily 

•  Review  scheduling  policy  to  determine  if  changes  are  needed. 

•  Redesign  the  system  (for  consistently  low  ratings). 

•  Accept  fewer  change  requests. 
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A  Glossary  of  Terms 


Acceptance  Review  A  review  of  a  software  product  by  developers  and  maintainers  to  deter¬ 
mine  if  the  product  satisfies  ali  originally  specified  requirements. 

Acceptance  Teat  Testing  led  by  the  client  or  QA  group  to  determine  whether  the  product 
satisfies  its  specifications  as  claimed  by  the  developer.[Sch90] 

Application  System  same  as  Information  System 

Availability  A  measure  of  the  degree  to  which  an  item  is  in  an  operable  and  committable  state 
at  the  start  of  a  mission  when  the  mission  is  called  for  at  a  random  point  in  time.[Dep82] 

Benchmark  Testing  Evaluation  of  the  system  performance  against  quantitative  requirement8.[Sch90] 

Change  Request  Review  Board  An  authority  responsible  for  evaluating  and  approving  re¬ 
quests  for  changes  to  a  software  product. 

Cohesion  A  measure  of  the  degree  of  the  functional  relatedness  within  program  units.  [Som89] 

Complexity  A  characteristic  of  the  software  interface  which  influences  the  resources  another 
system  will  expend  or  commit  while  interfacing  with  the  software.  [CDS86) 

Configuration  Management  The  process  of  identifying  and  defining  the  configuration  items 
(hardware/software  units)  in  a  system,  controlling  the  release  and  change  of  these  items 
throughout  the  system  life  cycle,  recording  and  reporting  the  status  of  configuration  items 
and  change  requests,  and  verifying  the  completeness  and  correctness  of  configuration  items.[IEE83] 

Consistency  The  extent  to  which  uniform  design  techniques  and  notation  are  used.  [War87] 

Coupling  A  measure  of  the  strength  of  interconnections  (dependencies)  between  program  units. 
[Som89] 

Error  Human  action  that  results  in  software  containing  a  fault.  Examples  include  omission  or 
misinterpretation  of  user  requirements  in  a  software  specification,  incorrect  translation  or 
omission  of  a  requirement  in  the  design  specification.  [IEE83] 

Failure  A  departure  of  program  operation  from  program  requirements. [IEE83] 

Failure  Rate  The  number  of  failures  of  an  item  per  measure-of-life  unit.[Dep82] 

Fault  A  manifestation  of  an  error  in  software.  A  fault,  if  encountered,  may  cause  a  faulure. 
Synonyniou:,  with  bug. 

Fourth  Generation  Language  (4GL)  A  computer  programming  langusige  that  provides  ab¬ 
stractions  of  data  and/ur  procedural  specifications  and  is  usually  suited  for  a  particular 
application  domain. 

Integration  Testing  Verify  that  the  modules  of  the  system  combine  correctly  in  order  to  achieve 
a  product  that  meets  its  specifications.  (Sch90) 


IS  (Information  Systems)  Organization  An  organied  collection  of  procedures,  personnel, 
and  resources  dedicated  to  support  a  portfolio  of  information  systems. 

Lines  of  Code  Lines  of  source  code,  not  including  comments. 

Maintainability  The  probability  that  an  item  will  be  retained  in,  or  restored  to,  a  specified 
condition  within  a  given  period  if  prescribed  procedures  and  resources  are  U8ed.[Dep82] 

Maintenance  AU  actions  required  to  retain  an  item  in,  or  restore  it  to,  a  specified  condition.[Dep82] 

Maintenance  Audit  An  organized  review  of  the  maintenance  organization. 

Maintenance  Escort  Participation  of  the  software  maintainer  in  software  system  development. 

Man/Machine  Interface  The  software  that  supports  the  interaction  between  the  user  and  the 
system. 

Measure  A  high-level  unit  of  specification  which  characterizes,  evaluates,  or  predicts  various 
aspects  of  software  life  cycle  processes  and  products. 

Metric  A  measurable  indication  of  some  aspect  of  a  system.  [DeM82]  A  quantification  of  a 
specific  feature  of  the  software  life  cycle  process  or  software  product. 

Modularity  A  characteristic  of  software  such  that  it  is  well-structured,  highly  cohesive,  and 
minimally  coupled.  (War87] 

New  Systems  Development  The  development  of  a  system  which  has  never  been  fielded. 

Object  Oriented  Design  Designing  a  system  in  terms  of  abstract  data  types  where  the  objects 
are  instantiations  of  the  data  types  and  new  data  types  can  be  defines  as  extensions  of 
previously  defined  types. 

Regression  Testing  Testing  the  system  against  previous  test  cases  to  ensure  that  the  function¬ 
ality  of  the  system  has  not  been  compromised  by  recent  changes  to  the  system.  [Sch90] 

Reliability  The  probability  that  an  item  will  perform  its  intended  function  for  a  specified  interval 
under  stated  condition8.[Dep82] 

Self- Descriptiveness  A  characteristic  of  software  that  enables  the  understanding  of  implemen¬ 
tation  of  software  functions.  [War87] 

Support  Staff  The  personnel  tasked  with  maintaining  an  information  system. 

Supportability  A  measure  of  the  adequacy  of  products,  resources,  and  procedures  to  facili¬ 
tate  the  support  activities  of  modifying  and  installing  software,  establishing  an  operational 
software  baseline,  and  meeting  user  requirements.  [PTH87] 

Testability  The  extent  to  which  software  facilitates  both  the  establishment  of  test  criteria  and 
the  evaluation  of  the  software  with  respect  to  those  criteria.  [IEE83] 

Throw-away  prototyping  Creating  a  prototype  as  part  of  system  design  and  then  “throwing 
away"  the  prototype  and  implementing  the  system  “from  scratch”  not  using  any  of  the 
source  code  from  the  prototype. 


Top'down  design  Designing  the  system  by  recursively  breaking  the  system  down  into  smaller 
components. 

Unit  Testing  Testing  of  individu2il  portions  of  the  system. 


B  Lbt  of  Acronyms 

AIRMICS  U.S.  Army  Institute  for  Research  in  Management  Information,  Communications, 
and  Computer  Science 

AMC  Army  Materiel  Command 

CCB  Change  Control  Board 

COE  Army  Corps  of  Engineers 

FORSCOM  Forces  Command 

HSC  Army  Health  Services  Command 

IS  Information  System 

ISC  Army  Information  Systems  Command 

LOC  Lines  of  Code 


C  System  Questionnaire 


This  appendix  contains  a  7  page  questionnaire  for  gathering  information  about  the  information 
system  in  order  to  calculate  the  Operational  Readiness  Measure.  The  questionnaire  should  be 
photocopied  and  distributed  to  selected  respondents. 


Software  Supportability  Qualitative  Assessment  Methodology 
Operational  Readiness  Questionnaire 

******************************************************************************* 
QUESTIONNAIRE  NUMBER:  _ 

Name  of  Information  System;  _ 

Software  and  Documentation  Information 


1.  What  is  the  size  of  the  system  source  code,  in  lines  of  code  (LOG)? 
_  lines  of  code 

2.  What  language (s)  is  the  software  written  in? 


3.  How  many  modules  (units  that  perform  single  functions  or  sets  of 
functions)  does  the  software  product  contain? 

number  of  modules 


4.  What  is  the  age  (measured  from  date  of  original  installation) 
of  the  software  product? 

age  of  system  (in  years) 


5.  How  long  has  your  organization  supported  this  software  product? 
_  length  of  support  (in  years) 


6 .  What  are  the  TOTAL  number  of  changes  that  have  been  made  to  this  product 
(software  and  associated  documentation)  during  the  time  you  have 
supported  it?  Include  both  Software  Change  Pac)cages  and  Emergency 
Change  Packages. 

_  total  number  of  changes 


7.  Does  the  software  contain  any  code  that  aids  in  debugging  the  software? 

_  yes 

no 
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********* ******************************************************* ****** ★★****★** 

Software  Supportability  Qualitative  Assessment  Methodology 
Operationcl  Readiness  Questionnaire 

****************************************************************************** 


8.  Is  there  any  documentation  explaining  the  overall  function  of  the 
software? 

_  yes 

no 


9.  Is  there  documentation  for  each  module  explaining  the  module's  function? 

_  yes 

no 


10.  Are  there  any  user's  manuals  explaining  the  use  of  this  software? 

_  yes 

no 


Maintenance  Information 

11.  For  what  amount  of  time  (how  many  hours)  during  the  month,  if  any,  is 
the  software  system  down  and  cannot  be  used? 

(hours)  down  time 

12.  What  is  the  average  number  of  maintenance  requests  per  month  received 
for  this  system? 

[Notes:  If  a  change  proposal  contains  several  requests,  count  each 
recjuest  separately. 

Count  ALL  requests,  even  those  that  no  actions  are  taken  on.] 
_  average  number  of  maintenance  requests  per  month 

13.  Approximately  how  many  of  the  above  maintenance  requests  (per  month) 
ultimately  result  in  some  change  being  made  to  the  software? 

_  percentage  of  requests  (per  month)  which  result  in  changes 

to  the  software 

14.  Approximately  what  percentage  of  the  maintenance  requests  FOR  WHICH 
YOU  PERFORM  ACTIONS  ON  are 

_  Small-scale  (affect  a  few  lines  of  code  at  most)? 

_  Medium-scale  (affect  several  functions  or  modules) ? 

_  Large-scale  (affect  all  or  a  large  portion  of  the  software)? 

100  %  TOTAL 
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Software  Supportabllity  Qualitative  Assessment  Methodology 
Operational  Readiness  Questionnaire 


15.  Approximately  what  percentage  of  the  maintenance  requests  FOR  WHICH 

YOU  PERFORM  ACTIONS  ON  are 

_  EMERGENCY  (require  immediate  attention  and  must  be  completed  as 

soon  as  possible  to  ensure  the  correct  operation  of  the  software) 

_  URGENT  (require  urgent  attention  —  more  so  than  normal  requests  - 

and  must  be  completed  within  a  relatively  short  period  of  time) 

_  NORMAL  (require  no  special  attention  and  can  be  completed  within 

the  usual  framework  of  support  procedures) 


100  %  TOTAL 


16.  What  percentage  of  ALL  maintenance  requests  you  receive... 

_  Are  for  corrections  to  faulty  software  components? 

_  Are  for  changes  (other  than  corrections)  or  enhancements  to  the 

software? 


100  %  TOTAL 


17.  What  percentage  (0-100%)  of  EMERGENCY  and  URGENT  requests  are  for 
corrections  to  faulty  software  components? 

_  percentage  of  EMERGENCY  and  URGENT  requests  that  are  corrections 


18.  ON  THE  AVERAGE,  what  percentage  (0-100%)  of  all  requests  require  more  time 
to  complete  than  is  originally  scheduled? 

_  percentage  of  all  requests  completed  behind  schedule 


19.  What  percentage  of  time  spent  maintaining  the  software  is  devoted  to 
testing  it? 

_  (%)  time  spent  on  testing 
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Software  Supportability  Qualitative  Assessment  Methodology 
Operational  Readiness  Questionnaire 
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User  Information 

20.  ON  THE  AVERAGE,  how  often  do  you  communicate  (either  formally  or 

informally)  with  a  TYPICAL  user  organization  using  this  information 
system?  Mark  the  one  appropriate  response  below. 

_  Several  times  a  day 

_  Once  or  twice  a  day 

_  At  least  weekly,  but  not  daily 

_  At  least  monthly,  but  not  weekly 

_  At  least  once  per  year,  but  not  monthly 

_  Less  than  once  per  year 


Current  Circumstances 


21.  How  many  people  in  your  support  organization  presently  maintain  this 
software  either  on  a  part-time  or  full-time  basis? 

(Indicate  the  number  in  each  category.) 

_  full-time  (number) 

_  part-time  (number) 


22.  AT  PRESENT  (NOT  on  the  average),  how  many  changes  of  all  types 

(including  corrections  and  enhancements)  are  there  to  be  implemented? 

_  number  of  changes  to  be  implemented 


23.  Of  the  eJDOve  changes  to  be  implemented,  what  percentage  (0-100%)  of 
these  changes  are  EMERGENCY  changes?  If  there  are  no  changes, 
answer  0% . 

_  percentage  of  current  changes  that  are  EMERGENCY 

24.  Of  the  changes  (from  #2)  to  be  implemented,  what  percentage  (0-100%) 
of  these  changes  are  for  CORRECTIONS  to  faulty  software  components? 
If  there  are  no  changes,  answer  0%. 

_  percentage  of  current  changes  that  are  CORRECTIONS 
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Software  Support ability  Qualitative  Assessment  Methodology 
Operational  Readiness  Questionnaire 

A********************************************************************** 


25.  Based  on  the  following  scale,  how  you  you  rate  the  estimated  effort 
needed  to  complete  changes  to  the  software  product  over  the  next 
month : 


0  -  Much  more  effort  than  average 

1  -  Somewhat  more  effort  than  average 

2  -  Average  effort 

3  “  Less  than  average  effort 

4  -  Much  less  than  average  effort 

5  -  No  effert  at  all  {no  changes  to  implement) 

answer  (0-5) 


Problem  Information 


26.  Overall,  in  your  judgment,  to  what  extent  are  (or  have  been)  the 
following  problems  in  maintaining  this  information  system? 

(Chec)c  the  appropriate  category.) 


No  Problem  At  All 


Somewhat  Minor  Problem 


Minor  Problem 


Somewhat  Major  Problem 


Major  Problem  | 


a.  Not  enough  people  to  support  this 
system. 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

b.  People  supporting  this  system  are 
not  trained  adequately. 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

c.  System  is  overly  large,  making 
support  difficult. 

1 

1 

1 

1 

1 

1 

i 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

d.  System  is  overly  complex,  making 
support  difficult. 

1 

I 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

e.  System  is  not  well-structured 
(written  in  "spaghetti  code") . 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

f .  Lack  of  system  modularization 
makes  changes  difficult  to 
implement . 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 
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*********************************************************************** 

Software  Supportability  Qualitative  Assessment  Methodology 
Operational  Readiness  Questionnaire 


26  (cont'd) 

No  Problem  At  All 

Somewhat  Minor  Problem  1 

_  I 

Minor  Problem  I  I 

_  I  I 

Somewhat  Major  Problem  1  I  ( 


Major  Problem  1 


g.  System  is  old  and  needs  to  be 
replaced. 


h.  System  documentation  is 
incomplete  or  confusing. 

i.  System  documentation  is  out-of- 
date. 

j .  Not  enough  time  is  spent  on 
testing  after  changes  are  made. 

k.  Software  repair  schedules  are 
hard  to  meet. 

l.  Overall,  there  are  more  change 
requests  submitted  for  this 
system  than  can  be  handled. 

m.  There  are  too  many  change 
requests  resulting  from  software 
bugs  (vs.  enhancement  requests). 

n.  There  are  too  many  emergency 
change  requests. 

o.  User  requirements  for  this 
system  change  frequently. 
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Software  Supportability  Qualitative  Assessment  Methodology 
Operational  Readiness  Questionnaire 


27.  Overall,  from  your  perspective,  to  what  extent  are  (or  have  been)  the 
problems  as  they  impact  on  the  ability  to  maintain  this  information 
system?  (Check  the  appropriate  category.) 


No  Problem  At  All 


Somewhat  Minor  Problem 


Minor  Problem 


Somewhat  Major  Problem 


Major  Problem  | 


a.  Skills  of  maintenance  programming 
personnel 

b.  Number  of  maintenance  programming 
personnel  available 

c.  Inadequate  hardware/software 
configurations  in  IS  Organization 

d.  Motivation  of  maintenance 
programming  personnel 

e.  Maintenance  programming 
productivity 

f.  Competing  demands  between  new 
systems  development  and 
maintenance 

g.  Budgetary  pressures 

h.  Meeting  scheduled  commitments 
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D  Scoring  Directions 

This  appendix  contains  the  following  two  items: 


1.  A  two- page  worksheet  for  recording  scores  from  the  operational  readiness  questionnaire. 

2.  Twenty  pages  of  directions  for  scoring  the  operational  readiness  questionnaire. 


Software  Supportability  Qualitative  Assessment  Methodology 
Operational  Readiness  Worksheet 


QUESTIONNAIRE  NUMBER  _ 

Name  of  Information  System: 


Question  Supportability  Reliability  Current  State  RANGE 


1. 

1  1  1  1  1  1  1  1  1  1 

INI 

1 

11  11 

0-5 

o 

^  • 

I  1  M  1  1  1  1  1  i 

Mil 

1 

11  11 

0-5 

3. 

1  1  1  I  1  1  1  1  1  1 

INI 

1 

MM 

0-5 

4  . 

1 1 1 1 1 1 1 1  n 

1  1  1  1 

1 

0-5 

5. 

1 1 1 1 1 1 1 1 1 1 

IN' 

1 

0-5 

6. 

i  1 1 1 1 1 1 1 1 1 

INI 

1 

11  11 

0-5 

7  . 

1  1  1  1  1  1  1  !  1  1 

INI 

1 

11  1  1 

0-2 

8. 

1  1  1  1  !  1  1  1  1  1 

MM 

1 

0-2 

9. 

1  1  1  !  M  1  1  M 

MM 

1 

11  11 

0-2 

10. 

1  I  1  1  1  1  1  1  1  1 

11  11 

1 

1  11  1 

0-2 

11. 

1  1  1  1  1  1  1  M  1 

MM 

1 

1  11  1 

0-10 

12. 

1  1  1  1  1  1  1  1  1  1 

1  1  1  1  1  1  1  1  1  1 

0-5 

13. 

1  1  1  1  1  1  1  1  1  1 

1  1  1  1  1  1  1  1  1  1 

0-10 

14. 

1  1  1  1  1  1  1  1  1  1 

MM 

1 

MM 

1-4 

15. 

1  1  1  1  1  1  1  1  1  ! 

MM 

1 

11  11 

1-4 

16. 

1  1  1  1  1  1  1  1  1  1 

MM 

1 

11  1  1 

0-20 

17. 

1  1  1  1  1  1  1  1  1  1 

MM 

1 

11  11 

0-5 

18.* 

1  1  1  1  1  1  1  1  1  1 

0-5 

19. 

1  1  M  i  1  1  1  i  1 

MM 

1 

11 1  1  1 

0-4 

20. 

i  1  1  1  1  1  1  1  1  1 

11  1  I 

1 

Mill 

0-10 

21. 

1  1  1  M  1  1  1  1  1 

1  1  1  1  1  1  i  1  1  1 

0-5 

22. 

i  1  N  1  1  1  I  1  1 

1  1  1  1  1  1  1  1  1  1 

0-20 

23. 

1  1  1  1  1  1  1  1  1  1 

MM 

1  1  M  11 

0-10 

24  .* 

1  1  1  1  1  1  1  1  1  1 

0-10 

25. 

1  1  1  1  1  1  1  1  1  1 

1  1  1  1  1  1  1  1  I  1 

0-10 

TOTALS 
THIS  PAGE 


(Carry  totals  over  to  next  page.) 


*  Enter  the  score  in  both  of  the  indicated  columns. 
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Software  Supportability  Qualitative  Assessment  Methodology 
Operational  Readiness  Worksheet 


Question  Supportability  Reliability  Current  State  RANGE 


TOTALS  FROM 
LAST  PAGE 


26a . 

i  1  i  1 1 1 1 1 1 1 

1  1  i 

1  1  1  1  1  1 

0-5 

b. 

1 1 11 1 1 1 1 1 1 

1  1  1 

MINI 

0-5 

c. 

1  1  1  1  M  1  1  1  1 

1  1  1 

1  M  1  1  1 

0-5 

d. 

1  1  1  1  1  1  1  1  1  1 

1  1  1 

1  1  1  1  1  1 

0-5 

e. 

i  1  1  1  1  1  1  1  1  1 

1  1  1 

1  M  1  1 

0-5 

f . 

i  i  1  1  1  1  1  1  1  1 

1  1  1 

1  1  1  1  1  1 

0-5 

g- 

1  1  1  1  1  1  1  1  1  1 

1  1  1 

1  1  1  1  1  1 

0-5 

h. 

1  1  1  1  1  1  1  1  1  1 

1  1  1 

1  1  1  1  1  1 

0-5 

i . 

1  I  1  1  1  1  I  1  1  1 

1  1  1 

1  1  1  1  1  1 

0-5 

j- 

1  1  1  1  i  1  1  1  1  1 

1  1  1 

MINI 

0-5 

k . 

1  1  I  1  1  1  1  1  1  1 

t  1  1  I  M  M  1  1 

0-5 

1. 

1  M  1  1  1  1  1  1  1 

M  1  1 1  1  1  1  1  1 

0-5 

m. 

1 1 1 1 1 1 1 1 1 1 

1  1  1 

1  1  1  1  1  1 

0-5 

n. 

1 1 1 1 1 1 1 1 1 1 

1  >  ) 

1  M  I  1  1 

0-5 

0. 

1 1 1 1 1 1 1 1 1 1 

1  1  1  1  1  1 

0-5 

27a. 

1  11  1  1  1  1  1 1 1 

1  1  1 

1  M  1  1  1 

0-5 

b. 

1  I  1  1  1  1  1  1  1  1 

1  1  1 

1  1  1  1  1  1 

0-5 

c. 

1  1  1  1  1  1  1  1  1  1 

1  1  1 

1  1  1  1  1  1 

0-5 

d. 

1  1  1  I  1  1  1  1  1  1 

1  1  1 

1  1  1  1  1  1 

0-5 

e. 

1  1  I  1  1  1  1  i  M 

I  1  1 

MINI 

0-5 

f . 

1  1  1  1  1  1  1  1 1  1 

1  1  1 

HUM 

0-5 

g- 

1  1  I  1  1  1  1  1  1  1 

1 1  1  1  1  1 

0-5 

h. 

i  1  1  1  1  1  1  1  1  1 

1  1  1 

1 1  1 1  1 1 

0-5 

TOTALS 

t  1  1  1  1  1  1  1  1  1 

1  1  1 

1  1  1  1  1  1 

1 1 1 1 1 1 1 1 1 1 

1  1  1 

1  1  1  1  1  1 

1 1 1 1 1 1 1 1 1 1 

1  1  I  1  1  1  1  1  1  1 

Software  Supportability  Qualitative  Assessment  Methodology 
Operational  Readiness  Questionnaire  -  Scoring  Directions 

QUESTIONNAIRE  NUMBER  _ 

Neune  of  Information  System:  _ 


1.  What  is  the  size  of  the  system  source  code,  in  lines  of  code  (LOG)? 
_  lines  of  code 


SCORING 


Calculate  the  score  utilizing  the  following  scale: 


System  Size 

At  least  But  less  than 


SCORE 


0 

10,000 

50,000 

100,000 

500,000 

1,000,000 


10,000  lines  of  code 
50,000 

100,000  " 

500,000  " 

1,000,000  ’• 


5 

4 

3 

2 

1 

0 


SCORE 


Software  Supportability  Qualitative  Assessment  Methodology 
Operational  Readiness  Questionnaire  -  Scoring  Directions 

*********************-********************************************************* 


2.  What  language (s)  is  the  software  written  in? 


SCORING 


Scoring  should  be  based  upon  the  following  scale: 


Number  of  languages  SCORE 


1  4 

2  3 

3  2 

4  1 

Greater  than  4  0 


Add  one  additional  point  to  the  score  if  at  least  half  of 
the  languages  are  high-level,  4th  generation  languages 
or  later. 


Examples  of  allowable  languages 


Language 


COBOL 

Assembly 

C 

Ada 

DBASE  III 
SQL 


Allowed? 


No 

No 

No 

Yes 

Yes 

Yes 


SCORE  - 
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Software  Supportability  Qualitative  Assessment  Methodology 
Operational  Readiness  Questionnaire  -  Scoring  Directions 


3.  How  many  modules  (units  that  perform  single  functions  or  sets  of 
functions)  does  the  software  product  contain? 

number  of  modules 


SCORING 


To  calculate  a  score  for  this  question,  you  need  the  answers 
for  both  this  question  and  question  number  one  (system  size 
in  lines  of  code) . 

Calculate  the  average  module  size,  in  lines  of  code,  by  dividing 
the  answer  in  number  one  by  the  answer  to  this  question.  Then 
assign  a  score  according  to  the  following  scale: 

Average  Module  Size 

At  Least  But  Less  Than  SCORE 


0 

500 

lines  of  code 

5 

500 

1,000 

II 

4 

1,000 

2,000 

II 

3 

2,000 

3,000 

II 

2 

3,000 

5,000 

11 

1 

5,000 

It 

0 

Exanrple:  If  the  information  system  contains  200,000  lines 

of  code  and  it  contains  300  modules,  then  the 
average  module  size  is: 

200,000  /  160  •=  1,250  lines  of  code. 

Thus,  we  would  assign  a  score  of  3. 


SCORE  - 
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Software  Supportability  Qualitative  Assessment  Methodology 
Operational  Readiness  Questionnaire  -  Scoring  Directions 

****************************************************************************** 


4 .  What  is  the  age  (measured  from  date  of  original  installation) 
of  the  software  product? 


age  of  system  (in  years) 


SCORING 


Compute  the  score  using  the  following  scale: 
System  Age 


At  Least  But  Less  Than 


0 

1 

3 

6 

8 

10 


1 
•  3 
6 
8 

10 


year 

years 


SCORE 

5 

4 

3 

2 

1 

0 


SCORE  - 


5.  How  long  has  your  organization  supported  this  software  product? 
_  length  of  support  (in  years) 

SCORING 


Compute  the  score  using  the  following  scale  (NOTE  THAT  THIS  SCALE 
IS  DIFFERENT  FROM  THE  ONE  IN  QUESTION  4) ; 


Length  of  Support 
At  Least  But  Less  Than 


SCORE 


0 

1 

3 

6 

8 

10 


1  year  0 

3  years  1 

6  "  2 

8  "  3 

10  "  4 

"  5 


SCORE  - 
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Software  Supportability  Qualitative  Assessment  Methodology 
Operational  Readiness  Questionnaire  -  Scoring  Directions 


6.  What  are  the  TOTAL  number  of  changes  that  have  been  made  to  this  product 
(software  and  associated  documentation)  during  the  time  you  have 
supported  it?  Include  both  Software  Change  Packages  and  Emergency 
Change  Packages . 

_  total  number  of  changes 


SCORING 


To  compute  the  score  for  this  question,  you  need  the  answers 
for  both  this  question  and  question  number  5  (length  of  support) . 

Calculate  the  average  number  of  changes  per  year  by  dividing 
the  answer  to  this  question  by  the  answer  in  number  5.  Then 
assign  a  score  according  to  the  following  scale: 

Average  Number  of 
Changes  Per  Year 

At  Least  But  Less  Than  SCORE 


0 

5 

changes  per  year 

5 

5 

10 

It 

4 

10 

50 

II 

3 

50 

100 

II 

2 

100 

500 

II 

1 

500 

II 

0 

Example:  If  the  information  system  has  been  supported  for 

5  years,  and  a  total  of  175  changer  have  been 
implemented  to  this  system,  then  the  average 
number  of  changes  is: 

175  /  5  -  35  changes  per  year. 

Thus,  we  would  assign  a  score  of  3. 


SCORE  - 
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Software  Supportability  Qualitative  Assessment  Methodology 
Operational  Readiness  Questionnaire  -  Scoring  Directions 


7.  Does  the  software  contain  any  code  that  aids  in  debugging  the  software? 

_  yes 

no 


8.  Is  there  any  documentation  explaining  the  overall  function  of  the 
software? 

_  yes 

no 


9.  Is  there  dociimentation  for  each  module  explaining  the  module's  function? 

_  yes 

no 


10.  Are  there  any  user's  manuals  explaining  the  use  of  this  software? 

_  yes 

no 


SCORING 


For  each  of  questions  7  through  10,  assign  a  score  of  2  points 
for  each  "yes"  answer  and  a  score  of  0  points  for  each  "no” 
answer.  For  the  four  questions  combined,  a  maximum  score  of 
8  points  is  possible. 


SCORE  - 
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Software  Supportability  Qualitative  Assessment  Methodology 
Operational  Readiness  Questionnaire  -  Scoring  Directions 

★★★*★★★*★★★★★★*★★★★★★★*★★★*★★***★★**★**★*★*★★**★****************★************* 
Maintenance  Information 


11.  For  what  amount  of  time  (how  many  hours)  during  the  month,  if  any,  is 
the  software  system  down  and  cannot  be  used,  on  the  average? 

_  (hours)  down  time 


SCORING 


Calculate  the  score  utilizing  the  following  scale: 


Down  time  per  month 


Least 

But  Less  Than 

SCORE 

0 

1 

hours 

10 

1 

2 

It 

8 

2 

3 

n 

•  6 

3 

5 

tt 

4 

5 

10 

tt 

2 

10 

ti 

0 

SCORE  - 
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Software  Suppoxtability  qualitative  Assessment  Methodology 
Operational  Readiness  Questionnaire  -  Scoring  Directions 


12.  What  is  the  average  number  of  maintenance  requests  per  month  received 
for  this  system? 


[Notes;  If  a  change  proposal  contains  several  requests,  count  each 
request  separately. 

Count  ALL  requests,  even  those  that  no  actions  are  taken  on.] 
_  average  number  of  maintenance  requests  per  month 


SCORING 


Calculate  the  score  utilizing  the  following  scale: 

Avg.  Number  of 
Maintenance  Requests 

At  Least  But  Less  Than  SCORE 


0 

1 

requests/month 

5 

1 

5 

It 

4 

5 

10 

It 

3 

10 

50 

ft 

2 

50 

100 

It 

1 

100 

It 

0 

SCORE  - 
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Software  Supportablllty  Qualitative  Assessment  Methodology 
Operational  Readiness  Questionnaire  -  Scoring  Directions 


13.  Approximately  how  many  of  the  above  maintenance  requests  (per  month) 
ultimately  result  in  some  change  being  made  to  the  software? 

_  number  of  requests  (per  month)  which  result  in  changes  to 

the  software 


SCORING 


Calculate  the  score  utilizing  the  following  scale: 


Avg.  Number  of 
Maintenance  Requests 


Least 

But  Less  Than 

SCORE 

0 

1 

requests /month 

10 

1 

5 

n 

6 

5 

10 

rt 

6 

10 

50 

ti 

4 

50 

100 

tt 

2 

100 

0 

SCORE  * 
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Software  Supportability  Qualitative  Assessment  Methodology 
Operational  Readiness  Questionnaire  -  Scoring  Directions 


14.  Approximately  what  percentage  of  the  maintenance  requests  FOR  WHICH 
YOU  PERFORM  ACTIONS  ON  are 

_  Small-scale  (affect  a  few  lines  of  code  at  most)? 

_  Medium-scale  (affect  several  functions  or  modules) ? 

_  Large-scale  (affect  all  or  a  large  portion  of  the  software) ? 

100  %  TOTAL 

SCORING 


For  each  percentage,  multiply  by  the  corresponding  weight 
as  shown  in  the  following  weight  table; 


Type  of  Action  WEIGHT 


Small-Scale  4 
Medium-Scale  3 
Large-Scale  1 


S\am  the  resulting  products,  divide  the  result  by  100, 

and  round  to  the  nearest  integer.  Maximum  score  is  4  points, 

minimum  score  is  1  point. 


Example:  If  the  percentage  for  the  various  types  of 

maintenance  requests  are  as  follows; 

45%  Small-scale  requests 
45%  Medium-scale  requests 
10%  Large-scale  requests 


The  calculation  is: 

(45  *  4)  +  (45  *•  3)  +  (10  *  1) 

_  -  3.25, 


100 

which  rounds  to  3.  Thus,  the  score  is  3. 


SCORE  - 
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Software  Supportability  Qualitative  Assessment  Methodology 
Operational  Readiness  Questionnaire  -  Scoring  Directions 


15.  Approximately  what  percentage  of  the  maintenance  requests  FOR  WHICH 

YOU  PERFORM  ACTIONS  ON  are 

_  EMERGENCY  (require  immediate  attention  and  must  be  completed  as 

soon  as  possible  to  ensure  the  correct  operation  of  the  software) 

_  URGENT  (require  urgent  attention  —  more  so  than  normal  requests 

and  must  be  completed  within  a  relatively  short  period  of  time) 

_  NORMAL  (recjuire  no  special  attention  and  can  be  completed  within 

the  usual  frameworJc  of  support  procedures) 


100  %  TOTAL 


SCORING 


For  each  percentage,  multiply  by  the  corresponding  weight 
as  shown  in  the  following  weight  table: 


Type  of  Recjuest  WEIGHT 


Emergency  1 
Urgent  2 
Normal  4 


Sum  the  resrlting  products,  divide  the  result  by  100, 

and  round  to  the  nearest  integer.  Maximum  score  is  4  points, 

minimum  score  is  1  point. 


Example:  If  the  percentage  for  the  various  types  of 

maintenance  requests  are  as  follows: 

10%  Emergency 
10%  Urgent 
80%  Normal 


The  calculation  is: 

(10  *  1)  +  (10  *  2)  +  (80  *  4) 
_  -  3.50, 

100 

which  rounds  to  4.  Thus,  the  score  is  4. 


SCORE  - 


Page  11 


Software  Support ability  Qualitative  Assessment  Methodology 
Operational  Readiness  Questionnaire  -  Scoring  Directions 


16.  What  percentage  of  ALL  maintenance  requests  you  receive... 

_  Are  for  corrections  to  faulty  software  components? 

_  Are  for  changes  (other  than  corrections)  or  enhancements  to  the 

software? 

100  %  TOTAL 

SCORING 


To  calculate  the  score,  divide  the  latter  percentage  figure 
(percentage  of  non-corrective  changes)  by  5  and  round  the  result 
to  the  nearest  integer.  Maximum  score  20  points,  minimum 
score  0  points.  (If  the  number  of  ALL  maintenance  requests 
you  receive  is  0,  assume  a  score  of  20) . 


Example:  30%  corrections 

70%  enhancements 

Calculation:  70/5-14. 

The  score  is  14. 

SCORE  - 


17.  What  percentage  (0-100%)  of  EMERGENCY  and  URGENT  requests  are  for 
corrections  to  faulty  software  components? 

_  percentage  of  EMERGENCY  and  URGENT  requests  that  are  corrections 


SCORING 

Calculate  the  score  as  follows: 

(100  -  percentage)  /  20  -  Unrounded  Score 

Round  the  result  to  obtain  a  score  between  0  and  5,  inclusive. 

Example:  65%  of  Emergency  and  Urgent  requests  are  correct" xons . 

Calculation:  (100  -  65)  /  20  -  1.75, 

Which  rounds  to  a  score  of  2 . 

SCORE  -  _ 
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Software  Supportability  Qualitative  Assessment  Methodology 
Operational  Readiness  Questionnaire  -  Scoring  Directions 


18.  ON  THE  AVERAGE,  what  percentage  (0-100%)  of  all  requests  require  more  time 
to  complete  than  is  originally  scheduled? 

_  percentage  of  all  requests  completed  behind  schedule 

SCORING 

The  calculation  used  to  obtain  this  score  is  exactly  the  same 
as  used  in  question  17.  That  is,  apply 

(100  -  percentage)  /  20 

and  round  the  result.  (Score  between  0  and  5,  inclusive). 

Example:  20%  of  requests  are  maintained  behind  schedule 

Calculation:  (100  -  20)  /  20  -  4.0, 

A  score  of  4. 

SCORE  -  _ 

19.  What  percentage  of  time  spent  maintaining  the  software  is  devoted  to 
testing  it? 

_  (%)  time  spent  on  testing 

SCORING 

To  calculate  the  score  for  this  question,  utilize  the  following 
scale : 

Percentage  Test  Time 


At  Least  But  less  Than  SCORE 


0%  10%  0 

10%  20%  1 

20%  30%  2 

30%  40%  3 

40%  4 


SCORE  - 
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Software  Supportability  Qualitative  Assessment  Methodology 
Operational  Readiness  Questionnaire  -  Scoring  Directions 


User  Information 


20.  ON  THE  AVERAGE,  how  often  do  you  communicate  (either  formally 
informally)  with  a  user  organization  using  this  information 
system?  Mark  the  one  appropriate  response  below. 

SCORE 


Several  times  a  day  10 
Once  or  twice  a  day  8 
At  least  weekly,  but  not  daily  6 
At  least  monthly,  but  not  weekly  4 
At  least  once  per  year,  but  not  monthly  2 
Less  than  once  per  year  0 


SCORING 


The  list  of  scores  corresponding  to  the  selected  item  is 
shown  above . 


SCORE  - 
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Software  Supportability  Qualitative  Assessment  Methodology 
Operational  Readiness  Questionnaire  -  Scoring  Directions 


Current  Circumstances 


21.  How  many  people  in  your  support  organization  presently  maintain  this 
software  either  on  a  part-time  or  full-time  basis? 

(Indicate  the  number  in  each  category.) 

_  full-time  (number) 

_  part-time  (number) 


SCORING 


Add  the  above  two  numbers  together.  Then,  find  the  corresponding 
score  in  the  following  table: 


TOTAL  Number  of 

Support  Personnel  SCORE 


0  0 

1  1 

2  2 

3-5  3 

6-10  4 

Greater  than  10  5 


SCORE  - 


22.  AT  PRESENT  (NOT  on  the  average),  how  many  changes  of  all  types 

(including  corrections  and  enhancements)  are  there  to  be  implemented? 

_  number  of  changes  to  be  implemented 


SCORING 


Calculate  the  score  utilizing  the  following  scale: 

Number  of 
Pending  Changes 

At  Least  But  Less  Than  SCORE 


0 

1 

1 

5 

5 

10 

10 

50 

50 

100 

100 

SCORE  - 


changes  20 

"  16 

"  12 

"  8 

"  4 

”  0 


1  C, 
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Software  Supportability  Qualitative  Assessment  Methodology 
Operational  Readiness  Questionnaire  -  Scoring  Directions 


23.  Of  the  above  changes  to  be  implemented,  what  percentage  (0-100%)  of 
these  changes  are  EMERGENCY  changes?  If  there  are  no  changes, 
answer  0% . 

_  percentage  of  current  changes  that  are  EMERGENCY 


SCORING 

To  calculate  the  score,  use  the  following  formula: 
(percentage)  /  10 

and  round  the  result  to  obtain  a  score  betwenn  0  and  10, 
inclusive . 


SCORE  - 


24.  Of  the  changes  (from  #22)  to  be  implemented,  what  percentage  (0-100%) 
of  these  changes  are  for  CORRECTIONS  to  faulty  software  components? 
If  there  are  no  changes,  answer  0%. 

_  percentage  of  current  changes  that  are  CORRECTIONS 


SCORING 

To  calculate  the  score,  use  the  following  formula: 
(percentage)  /  10 

and  round  the  result  to  obtain  a  score  between  0  and  10, 
inclusive . 


SCORE  - 
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Software  Supportability  Qualitative  Assessment  Methodology 
Operational  Readiness  Questionnaire  '  Scoring  Directions 


25.  Based  on  the  following  scale,  how  you  you  rate  the  estimated  effort 
needed  to  complete  changes  to  the  software  product  over  the  next 
month : 


0  Much  more  effort  than  average 

1  -  Somewhat  more  effort  than  average 

2  -  Average  effort 

3  -  Less  than  average  effort 

4  -  Much  less  than  average  effort 

5  -  No  effort  at  all  (no  changes  to  implement) 

answer  (0-5) 


SCORING 


To  obtain  the  score,  multiply  the  above  answer  by  2.  Thus, 
if  the  answer  above  is  4,  then  the  score  is  8.  (Maximum 
score  10  points,  minimum  score  0  points) . 


SCORE  « 
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****★*★*****★******★****★***★★***★*** ***************************^^***^** 

Software  Supportability  Qualitative  Assessment  Methodology 
Operational  Readiness  Questionnaire  -  Scoring  Directions 

★★**********★***★*****************************★***★★★★*★★*★★**★**★***** 


Problem  Information 


26.  Overall,  in  your  judgment,  to  what  extent  are  (or  have  been)  the 
following  problems  in  maintaining  this  information  system? 

(Check  the  appropriate  category.) 


No  Problem  At  All 


Somewhat  Minor  Problem 


Minor  Problem 


Somewhat  Major  Problem 


Major  Problem  I 


SCORE 


a.  Not  enough  people  to  support  this 
system. 

b.  People  supporting  this  system  are 
not  trained  adequately. 

c.  System  is  overly  large,  making 
support  difficult. 

d.  System  is  overly  complex,  making 
support  difficult. 

e.  System  is  not  well-structured 
(written  in  "spaghetti  code") . 

£.  Lack  of  system  modularization 
makes  changes  difficult  to 
implement . 

g.  System  is  old  and  needs  to  be 
replaced. 

h.  System  documentation  is 
incomplete  or  confusing. 

i.  System  documentation  is  out-of- 
date. 

j.  Not  enough  time  is  spent  on 
testing  after  changes  are  made. 
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Software  Support ability  Qualitative  Assessment  Methodology 
Operational  Readiness  Questionnaire  -  Scoring  Directions 


26  (cont'd) 


No  Problem  At  All 


Somewhat  Minor  Problem 


Minor  Problem 


Somewhat  Major  Problem 

— 

Major  Problem  | 


k.  Software  repair  schedules  are 
hard  to  meet . 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1.  Overall,  there  are  more  change 
requests  submitted  for  this 
system  than  can  be  handled. 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

m.  There  are  too  many  change 

requests  resulting  from  software 
bugs  (vs.  enhancement  requests). 

1 

1 

1 

1 

1 

1 

) 

1 

1 

I 

1 

1 

1 

1 

1 

1 

1 

1 

1 

n .  There  are  too  many  emergency 
change  requests. 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

o.  User  requirements  for  this 
system  change  frequently. 

1 

1 

1 

1 

1 

i 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

SCORING 


For  each  lettered  item,  score 

5  points  for  "No  problem  at  all" 

4  points  for  "Somewhat  minor  problem" 

3  points  for  "Minor  problem" 

1  point  for  "Somewhat  major  problem" 

0  points  for  "Major  problem" 
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Software  Support ability  Qualitative  Assessment  Methodology 
Operational  Readiness  Questionnaire  -  Scoring  Directions 


27.  Overall,  from  your  perspective,  to  what  extent  are  (or  have  been)  the 
problems  as  they  impact  on  the  ability  to  maintain  this  information 
system?  (Check  the  appropriate  category.)  • 


No  Problem  At  All 


Somewhat  Minor  Problem 


Minor  Problem 


Somewhat  Major  Problem 


Major  Problem  | 


SCORE 


a.  Skills  of  maintenance  programming 
personnel 

b.  Number  of  maintenance  programming 
personnel  available 

c.  Inadequate  hardware/software 
configurations  in  IS  Organization 

d.  Motivation  of  maintenance 
programming  personnel 

e.  Maintenance  programming 
productivity 

f.  Competing  demands  between  new 
systems  development  and 
maintenance 

g.  Budgetary  pressures 

h.  Meeting  scheduled  commitments 


SCORING 


For  each  lettered  item,  score 

5  points  for  "No  problem  at  all" 

4  points  for  "Somewhat  minor  problem" 
3  points  for  "Minor  problem" 

1  point  for  "Somewhat  major  problem" 
0  points  for  "Major  problem" 
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£  Operational  Readiness  Worksheet  -  Final  Results 


This  appendix  contains  a  one- page  worksheet  for  calculating  the  final  operational  readiness  mea¬ 
sure. 


Software  Supportability  Qualitative  Assessment  Methodology 
Operational  Readiness  Worksheet 


NAME  OF  SYSTEM 


1.  AVERAGE  of  the 
Total  Scores 

a.  SupportadDility 

b.  Reliability 

c.  Current  State 


2.  AVERAGE  Reliability  plus 
Current  State 
(lb  +  Ic) 


3 .  AVERAGE  Total  Score 
(la  +  lb  +  Ic) 


4 .  DIVIDE  Total  Score  above 
by  the  number  3  for 
Scaled  Total 


Example:  AVERAGE  Supportability  Score:  120 


AVERAGE  Reliability  Score:  60 

AVERAGE  Current  State  Score:  50 


Reliability  +  Current  State  110 
Total  Score  230 
Scaled  Total  (rounded)  77 
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